Fixing the First-Request Lag: Optimizing Azure Functions and Container Apps for AI Microservices
The first request after an idle period takes thirty seconds. Every request after it takes two hundred milliseconds. Your monitoring looks fine, your load tests pass, and your users are still complaining. This is cold start — and for AI microservices carrying model weights and a gigabyte of dependencies, it is dramatically worse than for ordinary APIs. This guide covers exactly where the seconds go and how to get them back.
HTTP/1.1 504 Gateway Timeout
Content-Type: application/json
{
"statusCode": 504,
"message": "The request timed out. The upstream server failed to respond
within the configured timeout period."
}
# Application Insights — the tell-tale pattern:
timestamp name duration resultCode
2026-07-11T09:02:14.117Z POST /api/embed 31842.6 ms 504 <-- first request
2026-07-11T09:02:51.402Z POST /api/embed 214.3 ms 200
2026-07-11T09:02:58.911Z POST /api/embed 198.7 ms 200
# Azure Container Apps — the slow-start container being killed before it is ready:
Liveness probe failed: Get "http://10.0.1.14:8000/health": context deadline exceeded
Container 'inference-api' was killed and is restarting (back-off restarting failed container)Symptom: One anomalously slow (or timed-out) request after every idle period; all subsequent requests are fast. Failure point: Client → Front Door / APIM → cold Function or Container App replica → model load. Default platform behaviour: Both Azure Functions (Consumption) and Container Apps (minReplicas: 0) deallocate idle compute to zero. The next request must pay the full cost of allocation, image pull, runtime boot, and model load before a single token is ever generated.
Cold start is the one performance problem that hides from every tool you use to find performance problems. It does not show up in load testing, because load testing keeps the service warm by definition. It does not show up in your P50 or P95, because it is a single request in a thousand. It shows up as a support ticket that says "the app is slow the first time I use it in the morning" — a complaint so vague it gets closed as unreproducible. And for AI microservices, where a container may be carrying a gigabyte of PyTorch and a half-gigabyte of model weights, the gap between a warm request and a cold one is not a few hundred milliseconds. It is the difference between 200ms and half a minute.
"Cold start" is not one event. It is five, in strict sequence, and you cannot fix what you have not attributed. Before changing a single setting, understand which stage is actually costing you — because the remedies are completely different and applying the wrong one buys you nothing but a larger bill.
| Stage | What happens | Who controls it | The lever |
|---|---|---|---|
| 1. Allocate | Platform selects a node and assigns compute to your app | Azure | Only avoidable by never scaling to zero (warm capacity) |
| 2. Image pull | Container layers not already cached on the node are downloaded | You | Image size, layer caching, registry locality, artifact streaming |
| 3. Runtime boot | The Functions host or container runtime starts the language worker | Azure (mostly) | Stay on a current runtime version; avoid deprecated models |
| 4. App init | Imports execute, frameworks initialize, model weights load into memory | You | Dependency pruning, lazy imports, externalized weights, load-once |
| 5. Probe pass | Health probe must succeed before traffic is routed to the replica | You | Startup probe with a realistic budget — or the platform kills you |
If your image is 4GB, no amount of lazy-loading in your Python will help — you are losing the time in stage 2. If your image is lean but you call SentenceTransformer(...) inside the request handler, no amount of image slimming will help — you are losing it in stage 4, on every single request. Measure first. Section 10 shows how to read the split out of Application Insights.
Microsoft's own Container Apps guidance is blunt about this: cold starts are most noticeable in scenarios involving large container images, complex application initialization, and ML/AI workloads. That is not a coincidence — an AI microservice is simultaneously all three. Here is precisely why the same platform that starts a Node API in a second takes forty for an inference endpoint.
- The dependency tree is enormous. A single import torch pulls in hundreds of megabytes of shared libraries. Add transformers, numpy, scipy, and a CUDA runtime, and your image is measured in gigabytes before you have written a line of application code. Every one of those megabytes is downloaded in stage 2.
- The model weights are a second payload. Baking a 400MB embedding model — or a multi-gigabyte LLM — into the container image doubles down on the problem: the weights inflate the image pull, and then get reserialized again in stage 4.
- Initialization does real work. Loading a model is not a config parse. It allocates memory, reserializes tensors, and may compile or warm up kernels. This is genuine CPU and I/O work that no amount of platform tuning removes — it can only be moved, cached, or done once.
- Inference is slow, so timeouts are tight and probes are dangerous. Teams set aggressive gateway timeouts because inference is already slow. A cold start then blows straight through them — producing the 504 at the top of this page rather than a merely slow response.
- The training image gets shipped to production. Images frequently transition from training to inference with only minimal tweaks, dragging along notebooks, dev tooling, test frameworks, and training-only dependencies that inference never touches.
These do not add — they compound. A fat image makes the pull slow. A slow pull delays the app-init. A slow app-init trips the default liveness probe. The probe kills the container. Container Apps restarts it — and pays the entire cost again, from image pull onward. What began as a thirty-second cold start becomes a container that never successfully starts at all. This is the restart loop in the error payload above, and it is why Section 9 exists.
Architectural Topology: Failing vs Remediated
Almost every laggy AI microservice is built the same way, and almost every fast one is built the same other way. This is the target state that the five fixes below construct, piece by piece.
| Component | Failing configuration (current) | Remediated configuration (fix) |
|---|---|---|
| Warm capacity | Scale-to-zero by default (minReplicas: 0, always-ready 0) | Always-ready instances / minReplicas: 1 on the user-facing path |
| Container image | Single-stage build, ~2–4GB, training deps included | Multi-stage build, slim base, inference-only deps |
| Model weights | Baked into the image layer | Azure Files / storage mount, read at startup |
| Model loading | Inside the request handler — paid on every call | Module/global scope — loaded once per instance |
| Health probes | Platform default liveness probe kills the slow starter | Explicit startup probe with a realistic failureThreshold |
| Scale rules | Scale out only at saturation; short cooldown | Scale earlier than capacity; cooldown that outlives inference |
| Hosting plan | Classic Consumption (no warm-capacity knob at all) | Flex Consumption or Container Apps, chosen deliberately |
The single highest-leverage decision is which host you are on, because it determines whether you have a warm-capacity lever at all. The classic Consumption plan does not expose one — you cannot configure an always-ready instance count on it, full stop. If you are fighting cold start on classic Consumption, you are fighting a battle the platform will not let you win; the fix is to move.
| Host | Warm-capacity lever | Fit for AI microservices |
|---|---|---|
| Consumption | None. Deallocates when idle; no always-ready setting | Poor. Fine for background/timer work where lag is invisible; wrong for a user-facing inference endpoint |
| Flex Consumption | Always-ready instances (default 0); per-function scaling; Azure Files mounts | Strong. Scales to zero like Consumption but lets you pin warm instances, and mounts model files without packaging them |
| Premium (Elastic) | Always-ready (minimumElasticInstanceCount) and pre-warmed instances | Good, but a higher fixed floor. Choose when you also need long executions or full VNet maturity |
| Container Apps | minReplicas ≥ 1; KEDA scale rules; storage mounts; serverless GPU | Strong. The right home when you need your own image, GPU inference, or non-HTTP event scaling |
Reach for Flex Consumption when your inference is a lightweight call — an embedding, a classification, an orchestration around Azure OpenAI — and you want the Functions programming model. Reach for Container Apps when you need to control the image itself, run a model locally on GPU, or scale on queues and events. Both give you a warm-capacity lever and both support storage mounts for weights; the choice is about packaging and control, not about cold start.
Warm capacity is not free, and no honest guide pretends otherwise. Always-ready instances bill continuously and count against your regional quota; minReplicas: 1 means paying for a replica that sits idle all night. The engineering question is not "how do I eliminate cold start" but "which paths must be warm, and what does that floor cost?" Keep a warm floor on the user-facing endpoint. Let batch workers, queue consumers, and dev/staging scale to zero.
This is the blunt instrument, and it is the correct first move for any user-facing inference endpoint. It does not make your cold start faster — it makes it not happen on the request path, by keeping at least one instance permanently initialized. Every other fix in this guide then reduces what a cold start costs when it does occur (during scale-out, deployment, or platform maintenance), which is why you still need them.
The classic workaround is a timer function that pings the app every few minutes to stop it going idle. It is a hack, it is unreliable (it keeps an instance warm, not necessarily the one that serves the next request), it does nothing for scale-out cold starts, and on Flex Consumption or Container Apps you now have a supported, deterministic setting that does the job properly. Use the real lever.
This attacks stage 2. Every megabyte in your container image is a megabyte that must be pulled across the network onto a cold node before your code runs. A production inference API should not be a multi-gigabyte image — and it usually only is because it inherited the training environment.
Audit what you are actually shipping
- Strip training-only dependencies. Inference does not need Jupyter, notebooks, test frameworks, linters, dataset libraries, or training loops. Images routinely go from training to inference with only minimal tweaks — audit and cut.
- Use the CPU build if you are not using a GPU. The CPU-only build of PyTorch is dramatically smaller than the default CUDA-enabled wheel. If your endpoint runs embeddings on CPU, the CUDA runtime is pure dead weight in every pull.
- Use a slim base image. python:3.12-slim over the full image; runtime base over SDK base for .NET.
- Multi-stage build. Compile and install in a build stage, copy only the resulting artefacts into a clean runtime stage. The build toolchain never ships.
- Order layers by volatility. Install dependencies before copying application code, so a code change re-uses the cached dependency layer instead of invalidating it.
Put your Azure Container Registry in the same region as your Container Apps environment — a cross-region pull adds latency to every cold start for no benefit. For large AI images, enable artifact streaming on ACR so the container can begin executing while layers are still being fetched, rather than waiting for the full download. Microsoft's Well-Architected guidance calls out lean images plus artifact streaming plus minimum replicas as the three levers for GPU cold starts specifically.
This is the AI-specific fix, and the one most teams miss. Both Flex Consumption and Container Apps let you mount an Azure Files share directly into the app, precisely so your code can reach large binaries and ML models without packaging them in your deployment. Pre-download the model to the storage account once; every instance then reads it from the mount instead of pulling it across the internet — or worse, carrying it inside the image.
The wins compound: the image shrinks (stage 2 collapses), the weights are no longer duplicated per image layer, and all instances share one copy of the reference data rather than each downloading their own.
Flex Consumption supports Azure Files mounts for exactly this purpose — keep large binaries and ML models out of the deployment package so deployments stay small and cold starts stay fast. Note the constraint: only SMB shares are supported (NFS is not), and mounts authenticate with a storage account access key. Put the storage account in the same region as the app, or you have simply relocated the latency rather than removed it.
This is the cheapest fix in the guide and the one that most often turns out to be the actual bug. If you load the model inside the request handler, you are not suffering from cold start at all — you are paying the model-load cost on every single request, warm or cold. Teams chase platform settings for weeks before noticing this.
Load the model in module/global scope. It then executes once, when the worker initializes, and persists in memory for the life of that instance. Every subsequent request on that instance reuses it for free.
Lazy-import the paths you rarely take
Module-scope loading is right for the model you always need. It is wrong for heavy libraries only used by optional code paths — importing a library your endpoint touches once a day still costs you at every init. Defer those imports into the function that actually needs them, so the common path stays lean.
Some frameworks defer real work until the first inference — allocating buffers, compiling kernels, or resolving lazy graph nodes. The result is that request #1 is still slow even though the model "loaded" at init. Fix it by running a single throwaway inference during initialisation (encode a dummy string, run one forward pass) so the first real request lands on a fully warmed model.
This is the section that turns a slow service into a broken one, and it catches almost everybody. Container Apps automatically configures a liveness probe when ingress is enabled. If your container takes a long time to start — and an AI container loading model weights certainly does — the platform can decide the container is unhealthy and kill it before it ever finishes starting.
The container then restarts, pays the full cold-start cost again, gets killed again, and you are in the back-off restart loop from the error payload at the top of this article. The service never becomes healthy. The fix is to declare an explicit startup probe with a budget that reflects how long your model actually takes to load.
The /ready endpoint must tell the truth. Have it return 200 only once the model object actually exists in memory — never a hard-coded 200 OK, which would route live traffic to a replica that cannot yet serve it.
Scale out before saturation. If a replica handles 20 concurrent requests comfortably, set the threshold to 10 — new replicas then start warming before you are overwhelmed, rather than after. KEDA cannot compensate for a slow image pull, so it needs a head start.
Set a cooldown that outlives your inference. LLM calls can run 30+ seconds. A scale-in that fires mid-request kills the replica and the request with it. Set cooldownPeriod comfortably longer than your slowest expected inference.
Validation & Verification: Confirm the Fix
Cold start is defined by idleness, so you cannot validate the fix by hammering the endpoint — load keeps it warm and hides the very bug you are testing. You must deliberately reproduce the idle condition, then measure the first request. Three steps.
Warm capacity removes cold start from the idle path — it does not abolish cold start. A new replica created by a scale-out, a deployment, or platform maintenance still starts cold. That is expected and healthy. The success criteria are: no cold start on the first request after idle (Fix 1), and a materially cheaper cold start when one does occur (Fixes 2–5), with no restart loops. If you only apply Fix 1, your users are fine at 9am and your next scale-out event still times out.
Key Takeaways
Frequently Asked Questions
Related FAVRITE Articles
- How to Fix Azure OpenAI Token Limits: Architectural Patterns for High-Throughput Apps
- Deploy an AI Chatbot on Azure OpenAI and App Service
- AKS CrashLoopBackOff, Pending Pods, and NotReady Nodes
- ADLS Gen2 vs Blob Storage for AI Workloads